Systems and Methods for Visual Processing of Spectrograms to Generate Haptic Effects

ABSTRACT

Systems and methods for visual processing of spectrograms to generate haptic effects are disclosed. In one embodiment, a signal comprising at least an audio signal is received. One or more spectrograms may be generated based at least in part on the received signal. One or more haptic effects may be determined based at least in part on the spectrogram. For example, a generated spectrogram may be a two-dimensional image and this image can be analyzed to determine one or more haptic effects. Once a haptic effect has been determined, one or more haptic output signals can be generated. A generated haptic output signal may be output to one or more haptic output devices.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/874,933, filed Sep. 6, 2013, entitled “Audio to Haptics,” theentirety of which is hereby incorporated by reference.

FIELD

The present disclosure relates generally to systems and methods forvisual processing of spectrograms to generate haptic effects.

BACKGROUND

Traditionally, mechanical buttons have provided physical tactilesensations to the users of handheld mobile devices. However, with theincrease in popularity of touch-sensitive surfaces (e.g., touch screens)on these devices, especially on mobile phones, the mechanical buttonsare no longer present. Instead, haptic output devices may be included insuch devices to output haptic effects that alert the user to variousevents.

SUMMARY

Embodiments provide systems and methods for visual processing ofspectrograms to generate haptic effects. For example, one disclosedmethod comprises receiving a signal. In some embodiments, the signalcomprises an audio signal, a video signal, an acceleration signal, avelocity signal, a temperature signal, another suitable signal, or acombination thereof.

In some embodiments, the method comprises receiving a signal; generatinga spectrogram based at least in part on the signal; determining a hapticeffect based at least in part on the spectrogram; generating a hapticoutput signal based at least in part on the haptic effect, the hapticoutput signal configured to cause a haptic output device to output thehaptic effect; and outputting the haptic output signal.

In another embodiment, a computer-readable medium comprises program codefor: receiving a signal; generating a spectrogram based at least in parton the signal; determining a haptic effect based at least in part on thespectrogram; generating a haptic output signal based at least in part onthe haptic effect, the haptic output signal configured to cause a hapticoutput device to output the haptic effect; and outputting the hapticoutput signal.

In another embodiment, a system comprises: an input device; a hapticoutput device; and a processor in communication with the input deviceand the haptic output device. In this embodiment, the processor isconfigured for: receiving a signal from the input device; generating aspectrogram based at least in part on the signal; determining a hapticeffect based at least in part on the spectrogram; generating a hapticoutput signal based at least in part on the haptic effect, the hapticoutput signal configured to cause the haptic output device to output thehaptic effect; and outputting the haptic output signal to the hapticoutput device.

These illustrative embodiments are mentioned not to limit or define theinvention, but rather to provide examples to aid understanding thereof.Illustrative embodiments are discussed in the Detailed Description,which provides further description of the invention. Advantages offeredby various embodiments of this invention may be further understood byexamining this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate one or more examples ofembodiments and, together with the description of example embodiments,serve to explain the principles and implementations of the embodiments.

FIG. 1 illustrates an electronic device for visual processing ofspectrograms to generate haptic effects in accordance with anembodiment;

FIG. 2 illustrates an electronic device for visual processing ofspectrograms to generate haptic effects in accordance with anembodiment;

FIG. 3 illustrates a system diagram depicting computing devices forvisual processing of spectrograms to generate haptic effects inaccordance with an embodiment;

FIG. 4 illustrates a flow chart directed to a method of visualprocessing of spectrograms to generate haptic effects in accordance withan embodiment;

FIG. 5 illustrates aspects of visual processing of spectrograms togenerate haptic effects in accordance with an embodiment; and

FIG. 6 illustrates aspects of visual processing of spectrograms togenerate haptic effects in accordance with an embodiment.

DETAILED DESCRIPTION

Example embodiments are described herein in the context of systems andmethods for visual processing of spectrograms to generate hapticeffects. Those of ordinary skill in the art will realize that thefollowing description is illustrative only and is not intended to be inany way limiting. Other embodiments will readily suggest themselves tosuch skilled persons having the benefit of this disclosure. Referencewill now be made in detail to implementations of example embodiments asillustrated in the accompanying drawings. The same reference indicatorswill be used throughout the drawings and the following description torefer to the same or like items.

In the interest of clarity, not all of the routine features of theimplementations described herein are shown and described. It will, ofcourse, be appreciated that in the development of any such actualimplementation, numerous implementation-specific decisions must be madein order to achieve the developer's specific goals, such as compliancewith application- and business-related constraints, and that thesespecific goals will vary from one implementation to another and from onedeveloper to another.

Illustrative Device & Embodiment

FIG. 1 illustrates an electronic device 100 for visual processing ofspectrograms to generate haptic effect. In some embodiments, electronicdevice 100 receives an audio signal. For example, electronic device 100may receive an audio signal for a song to be played on the electronicdevice 100. The electronic device 100 can receive the song from memoryor from another device, such as from a server through the Internet. Oncethe electronic device 100 receives the audio signal, the electronicdevice 100 generates one or more spectrograms based on the receivedaudio signal. The electronic device 100 may generate a spectrogram bydetermining a Fourier transform of a time window, such as 10 ms, of thetime domain of the received audio signal over the entire audio signal.As another example, the electronic device may generate one spectrogramhaving a horizontal axis representing time, a vertical axis representinga first frequency range (e.g., 0 Hz to 100 Hz) with the magnitude at aparticular frequency being represented by a color intensity and anotherspectrogram having a horizontal axis representing time, a vertical axisrepresenting a second a second frequency range (e.g., 100 Hz to 1 KHz)with the magnitude at a particular frequency being represented by acolor intensity.

Once the spectrogram(s) have been generated, one or more haptic effectsmay be determined by the electronic device 100 by analyzing thegenerated spectrogram(s). For example, if a spectrogram indicatesregions that are spaced apart at regular intervals, then the electronicdevice 100 may determine that each of the regions is a beat anddetermine a haptic effect corresponding to a beat. For example, a hapticeffect having a vibration of limited duration and a relatively highmagnitude that is configured to mimic a beat may be determined. Asanother example, changes between regions in a spectrogram may indicate atransition from one type of sound to another and, thus, the electronicdevice 100 may determine a haptic effect before the transition and adifferent haptic effect during and/or after the transition. Once thehaptic effect(s) have been determined, the electronic device 100 cangenerate haptic output signal(s). The haptic output signals can beoutput by the electronic device 100 to one or more haptic output devicesthat output the determined haptic effect(s). Numerous other embodimentsare disclosed herein and variations are within the scope of thisdisclosure.

This illustrative example is given to introduce the reader to thegeneral subject matter discussed herein. The invention is not limited tothis example. The following sections describe various additionalnon-limiting embodiments and examples of devices, systems, and methodsfor parameter modification of haptic effects.

Illustrative Device

FIG. 2 illustrates an electronic device 200 for visual processing ofspectrograms to generate haptic effects according to one embodiment. Inthe embodiment shown in FIG. 2, the electronic device 200 comprises ahousing 205, a processor 210, a memory 220, a touch-sensitive display230, a haptic output device 240, a communication interface 250, and asensor 270. In addition, the electronic device 200 is in communicationwith haptic output device 260, which may be optionally coupled to orincorporated into some embodiments. The processor 210 is incommunication with the memory 220 and, in this embodiment, both theprocessor 210 and the memory 220 are disposed within the housing 205.The touch-sensitive display 230, which comprises or is in communicationwith a touch-sensitive surface, is partially disposed within the housing205 such that at least a portion of the touch-sensitive display 230 isexposed to a user of the electronic device 200. In some embodiments, thetouch-sensitive display 230 may not be disposed within the housing 205.For example, the electronic device 200 may be connected to or otherwisecommunicate with a touch-sensitive display 230 disposed within aseparate housing. In some embodiments, the housing 205 may comprise twohousings that are slidably coupled to each other, pivotably coupled toeach other, or releasably coupled to each other. In other embodiments,the housing 205 may comprise any number of housings.

In the embodiment shown in FIG. 2, the touch-sensitive display 230 is incommunication with the processor 210 and is configured to providesignals to the processor 210 and/or the memory 220 and to receivesignals from the processor 210 and/or memory 220. The memory 220 isconfigured to store program code or data, or both, for use by theprocessor 210, which is configured to execute program code stored inmemory 220 and to transmit signals to and receive signals from thetouch-sensitive display 230. In the embodiment shown in FIG. 2, theprocessor 210 is in communication with the communication interface 250and is configured to receive signals from the communication interface250 and to output signals to the communication interface 250 tocommunicate with other components or devices such as one or moreelectronic devices. In addition, the processor 210 is in communicationwith haptic output device 240 and haptic output device 260 and isfurther configured to output signals to cause haptic output device 240or haptic output device 260, or both, to output one or more hapticeffects.

Furthermore, the processor 210 is in communication with sensor 270 andis configured to receive signals from sensor 270. For example, processor210 may receive one or more signals from sensor 270 corresponding withone or more interactions with the electronic device 200. For example,one or more sensor signals may be received by processor 210 from sensor270 when a user of the electronic device 200 shakes the device 200. Asanother example, one or more sensor signals can be received by processor210 from sensor 270 when a user presses a location on thetouch-sensitive display 230 and/or when a user makes a gesture ontouch-sensitive display 230. In some embodiments, processor 210 canreceive sensor information from one or more sensors, such as sensor 270,to derive or otherwise determine one or more interactions. Interactionscan include, but are not limited to a contact, a series of contacts, agesture, a contact pressure above a predetermined threshold, a contactpressure below a predetermined threshold, a contact on a flexibletouch-sensitive surface such as a flexible touch-screen display, avibration, a shake, any other suitable interaction, or a combinationthereof.

In some embodiments, processor 210 receives one or more sensor signalsfrom one or more input devices integrated into the electronic device200, connected to the electronic device 200, and/or in communicationwith the electronic device 200. For example, the processor 210 mayreceive one or more sensor signals from a touch-sensitive surface of thetouch-sensitive display 230. As another example, the processor 210 mayreceive one or more sensor signals from an input device such as akeyboard, a mouse, a touchpad, a trackball, a microphone, atouch-sensitive surface, and/or another suitable input device that isintegrated into the electronic device 200, connected to the electronicdevice 200, and/or in communication with the electronic device 200. Asensor signal may comprise information such as one or more contacts,locations, pressures, gestures, key presses, and/or other informationindicating how a user is interacting with one or more input devices. Insome embodiments, a sensor signal and/or a received signal comprises anaudio signal, a video signal, an acceleration signal, a velocity signal,a temperature signal, another suitable signal, or a combination thereof.Numerous other embodiments are disclosed herein and variations arewithin the scope of this disclosure.

The processor 210 may then utilize the information it receives from oneor more sensors (such as sensor 270), one or more communicationinterfaces (such as communication interface 250), memory (such as memory220), and/or another suitable input device such as a an audio inputdevice and/or an audio/video input device to generate one or morespectrograms. For example, audio received from a microphone may be usedto generate one or more spectrograms. As another example, an audio filereceived from memory may be used to generate one or more spectrograms.In some embodiments, audio received from another device through acommunication interface is used to generate one or more spectrograms. Agenerated spectrogram may include a first axis representing frequencies,a second axis representing time, and intensities (such as colorintensities or greyscale intensities) for the frequencies at varioustimes. In some embodiments, the processor 210 analyzes at least aportion of at least one generated spectrogram to determine one or morehaptic effects. For example, the processor 210 may analyze a spectrogramto determine whether a pattern can be detected in the spectrogram. Insome embodiments where a pattern is detected, the processor 210 maydetermine that a haptic effect corresponds with the pattern and shouldbe output. In some embodiments, a pattern (or lack thereof) may indicatethat a particular haptic effect should be output, that a particularhaptic effect should stop being output, or that a haptic effect shouldbe changed from one haptic effect to another haptic effect. Numerousother embodiments are disclosed herein and variations are within thescope of this disclosure.

Once the processor 210 determines one or more effects that should beoutput, the processor 210 can generate one or more output signals. Forexample, in one embodiment, one or more output signals are generated byprocessor 210 in response to determining a pattern in a spectrogram. Asanother example, one or more output signals can be generated byprocessor 210 if part of a generated spectrogram indicates that apattern is ending. Numerous other embodiments are disclosed herein andvariations are within the scope of this disclosure.

The device illustrated in FIG. 2 is merely illustrative, and in variousother embodiments, the electronic device 200 may comprise or be incommunication with fewer or additional components and/or devices thanshown in FIG. 2. For example, other user input devices such as a mouse,a keyboard, a camera and/or other input device(s) may be comprisedwithin the electronic device 200 or be in communication with theelectronic device 200. As another example, electronic device 200 maycomprise or otherwise be in communication with one or more sensorsand/or one or more haptic output devices. In another example, electronicdevice 200 may not comprise a communication interface 250 in oneembodiment. As yet another example, electronic device 200 may not be incommunication with haptic output device 260 in an embodiment. Numerousother embodiments are disclosed herein and variations are within thescope of this disclosure.

Various other components may also be modified. For example, in someembodiments, sensor 270 is partially or fully disposed within housing205. As another example, sensor 270 may be disposed within the housing205 of the electronic device 200. In one embodiment, the electronicdevice 200 is not in communication with haptic output device 260 anddoes not comprise communication interface 250. In another embodiment,the electronic device 200 does not comprise a touch-sensitive display230 or a communication interface 250, but comprises a touch-sensitivesurface and is in communication with an external display. In otherembodiments, the electronic device 200 may not comprise or be incommunication with a haptic output device at all. Thus, in variousembodiments, the electronic device 200 may comprise or be incommunication with any number of components, such as in the variousembodiments disclosed herein as well as variations that would beapparent to one of skill in the art.

The electronic device 200 can be any device that is capable of receivinguser input. For example, the electronic device 200 in FIG. 2 includes atouch-sensitive display 230 that comprises a touch-sensitive surface. Insome embodiments, a touch-sensitive surface may be overlaid on thetouch-sensitive display 230. In other embodiments, the electronic device200 may comprise or be in communication with a display and a separatetouch-sensitive surface. In still other embodiments, the electronicdevice 200 may comprise or be in communication with a display and maycomprise or be in communication with other user input devices, such as amouse, a keyboard, buttons, knobs, slider controls, switches, wheels,rollers, other manipulanda, or a combination thereof.

In some embodiments, one or more touch-sensitive surfaces may beincluded on or disposed within one or more sides of the electronicdevice 200. For example, in one embodiment, a touch-sensitive surface isdisposed within or comprises a rear surface of the electronic device200. In another embodiment, a first touch-sensitive surface is disposedwithin or comprises a rear surface of the electronic device 200, and asecond touch-sensitive surface is disposed within or comprises a sidesurface of the electronic device 200. In some embodiments, theelectronic device 200 may comprise two or more housing components, suchas in a clamshell arrangement or in a slidable arrangement. For example,one embodiment comprises an electronic device 200 having a clamshellconfiguration with a touch-sensitive display disposed in each of theportions of the clamshell. Furthermore, in some embodiments where theelectronic device 200 comprises at least one touch-sensitive surface onone or more sides of the electronic device 200 or in some embodimentswhere the electronic device 200 is in communication with an externaltouch-sensitive surface, the display 230 may or may not comprise atouch-sensitive surface. In some embodiments, one or moretouch-sensitive surfaces may have a flexible touch-sensitive surface. Insuch embodiments, a user may be able to bend or otherwise deform theflexible touch-sensitive surface as a method of input. For example, inone embodiment, an electronic device has a flexible touch-screen displayand a user can deform the flexible touch-screen display by pressinglocations on the flexible touch-screen display to input information tothe electronic device. In other embodiments, one or more touch-sensitivesurfaces may be rigid. In various embodiments, the electronic device 200may comprise both flexible and rigid touch-sensitive surfaces.

The housing 205 of the electronic device 200 shown in FIG. 2 providesprotection for at least some of the components electronic device 200.For example, the housing 205 may be a plastic casing that protects theprocessor 210 and memory 220 from foreign articles such as rain. In someembodiments, the housing 205 protects the components in the housing 205from damage if the electronic device 200 is dropped by a user. Thehousing 205 can be made of any suitable material including but notlimited to plastics, rubbers, or metals. Various embodiments maycomprise different types of housings or a plurality of housings. Forexample, in some embodiments, electronic device 200 may be a portabledevice, handheld device, toy, gaming console, handheld video gamesystem, gamepad, game, controller, desktop computer, portablemultifunction device such as a cell phone, smartphone, personal digitalassistant (PDA), eReader, portable reading device, handheld readingdevice, laptop, tablet computer, digital music player, remote control,medical instrument, etc. In some embodiments, the electronic device 200may be embedded in another device such as a vehicle, wrist watch, otherjewelry, arm band, gloves, glasses, etc. Thus, in some embodiments, theelectronic device 200 is wearable. In an embodiment, the electronicdevice 200 is embedded in another device such as, for example, theconsole of a car or a steering wheel. Numerous other embodiments aredisclosed herein and variations are within the scope of this disclosure.

In the embodiment shown in FIG. 2, the touch-sensitive display 230provides a mechanism for a user to interact with the electronic device200. For example, the touch-sensitive display 230 detects the locationor pressure, or both, of a user's finger in response to a user hoveringover, touching, or pressing the touch-sensitive display 230 (all ofwhich may be referred to as a contact in this disclosure). In oneembodiment, a contact can occur through the use of a camera. Forexample, a camera may be used to track a viewer's eye movements as thereader views the content displayed on the display 230 of the electronicdevice 200. In this embodiment, haptic effects may be triggered based atleast in part on the viewer's eye movements. For example, a hapticeffect may be output when a determination is made that the viewer isviewing content at a particular location of the display 230. In someembodiments, the touch-sensitive display 230 may comprise, be connectedwith, or otherwise be in communication with one or more sensors thatdetermine the location, pressure, a size of a contact patch, or any ofthese, of one or more contacts on the touch-sensitive display 230.

For example, in one embodiment, the touch-sensitive display 230comprises or is in communication with a mutual capacitance system. Inanother embodiment, the touch-sensitive display 230 comprises or is incommunication with an absolute capacitance system. In some embodiments,the touch-sensitive display 230 may comprise or be in communication witha resistive panel, a capacitive panel, infrared LEDs, photodetectors,image sensors, optical cameras, or a combination thereof. Thus, thetouch-sensitive display 230 may incorporate any suitable technology todetermine a contact on a touch-sensitive surface such as, for example,resistive, capacitive, infrared, optical, thermal, dispersive signal, oracoustic pulse technologies, or a combination thereof. In someembodiments, a determined haptic effect is modified or otherwiseconfigured based at least in part on interactions and/or otherinformation received from one or more sensors that can be used todetermine one or more interactions. For example, an intensity parameterof a haptic effect may be increased or decreased based on one or moreinteractions. In some embodiments, the display is not touch-sensitive.In other embodiments, the electronic device 200 does not have a display.

In the embodiment shown in FIG. 2, haptic output devices 240 and 260 arein communication with the processor 210 and are configured to provideone or more haptic effects. For example, in one embodiment, when anactuation signal is provided to haptic output device 240, haptic outputdevice 260, or both, by the processor 210, the respective haptic outputdevice(s) 240, 260 outputs a haptic effect based on the actuationsignal. For example, in the embodiment shown, the processor 210 isconfigured to transmit a haptic output signal to haptic output device240 comprising an analog drive signal. In some embodiments, theprocessor 210 is configured to transmit a command to haptic outputdevice 260, wherein the command includes parameters to be used togenerate an appropriate drive signal to cause the haptic output device260 to output the haptic effect. In other embodiments, different signalsand different signal types may be sent to each of one or more hapticoutput devices. For example, in some embodiments, a processor maytransmit low-level drive signals to drive a haptic output device tooutput a haptic effect. Such a drive signal may be amplified by anamplifier or may be converted from a digital to an analog signal, orfrom an analog to a digital signal using suitable processors orcircuitry to accommodate the particular haptic output device beingdriven.

A haptic output device, such as haptic output devices 240 or 260, can beany component or collection of components that is capable of outputtingone or more haptic effects. For example, a haptic output device can beone of various types including, but not limited to, an eccentricrotational mass (ERM) actuator, a linear resonant actuator (LRA), apiezoelectric actuator, a voice coil actuator, an electro-active polymer(EAP) actuator, a memory shape alloy, a pager, a DC motor, an AC motor,a moving magnet actuator, an E-core actuator, a smartgel, anelectrostatic actuator, an electrotactile actuator, a deformablesurface, an electrostatic friction (ESF) device, an ultrasonic friction(USF) device, or any other haptic output device or collection ofcomponents that perform the functions of a haptic output device or thatare capable of outputting a haptic effect. Multiple haptic outputdevices or different-sized haptic output devices may be used to providea range of vibrational frequencies, which may be actuated individuallyor simultaneously. Various embodiments may include a single or multiplehaptic output devices and may have the same type or a combination ofdifferent types of haptic output devices. In some embodiments, one ormore haptic output devices are directly or indirectly in communicationwith electronic device, such as via wired or wireless communication. Inone embodiment, the electronic device can be placed in a vehicle or isintegrated into a vehicle and one or more haptic output devices areembedded into the vehicle. For example, one or more haptic outputdevices may be embedded in a seat, steering wheel, pedal, etc. of thevehicle. In some embodiments, instead of having haptic output device 240and/or haptic output device 260 or in addition to having haptic outputdevice 240 and/or haptic output device 260, the electronic device 200has one or more other output devices. For example, the electronic device200 may have a speaker and/or a display. In one embodiment, theelectronic device 200 has one or more haptic output devices, one or morespeakers, and one or more displays. Numerous other embodiments aredisclosed herein and variations are within the scope of this disclosure.

In various embodiments, one or more haptic effects may be produced inany number of ways or in a combination of ways. For example, in oneembodiment, one or more vibrations may be used to produce a hapticeffect, such as by rotating an eccentric mass or by linearly oscillatinga mass. In some such embodiments, the haptic effect may be configured toimpart a vibration to the entire electronic device or to only onesurface or a limited part of the electronic device. In anotherembodiment, friction between two or more components or friction betweenat least one component and at least one contact may be used to produce ahaptic effect, such as by applying a brake to a moving component, suchas to provide resistance to movement of a component or to provide atorque. In order to generate vibration effects, many devices utilizesome type of actuator and/or other haptic output device. Known hapticoutput devices used for this purpose include an electromagnetic actuatorsuch as an Eccentric Rotating Mass (“ERM”) in which an eccentric mass ismoved by a motor, a Linear Resonant Actuator (“LRA”) in which a massattached to a spring is driven back and forth, or a “smart material”such as piezoelectric, electro-active polymers or shape memory alloys.

In other embodiments, deformation of one or more components can be usedto produce a haptic effect. For example, one or more haptic effects maybe output to change the shape of a surface or a coefficient of frictionof a surface. In an embodiment, one or more haptic effects are producedby creating electrostatic forces and/or ultrasonic forces that are usedto change friction on a surface. In other embodiments, an array oftransparent deforming elements may be used to produce a haptic effect,such as one or more areas comprising a smartgel. Haptic output devicesalso broadly include non-mechanical or non-vibratory devices such asthose that use electrostatic friction (ESF), ultrasonic surface friction(USF), or those that induce acoustic radiation pressure with anultrasonic haptic transducer, or those that use a haptic substrate and aflexible or deformable surface, or those that provide projected hapticoutput such as a puff of air using an air jet, and so on. In someembodiments, a haptic effect is a kinesthetic effect.

In FIG. 2, the communication interface 250 is in communication with theprocessor 210 and provides wired or wireless communications, from theelectronic device 200 to other components or other devices. For example,the communication interface 250 may provide wireless communicationsbetween the electronic device 200 and a wireless sensor or a wirelessactuation device. In some embodiments, the communication interface 250may provide communications to one or more other devices, such as anotherelectronic device 200, to allow users to interact with each other attheir respective devices. The communication interface 250 can be anycomponent or collection of components that enables the multi-pressuretouch-sensitive input electronic device 200 to communicate with anothercomponent or device. For example, the communication interface 250 maycomprise a PCI network adapter, a USB network adapter, or an Ethernetadapter. The communication interface 250 may communicate using wirelessEthernet, including 802.11a, g, b, or n standards. In one embodiment,the communication interface 250 can communicate using Radio Frequency(RF), Bluetooth, CDMA, TDMA, FDMA, GSM, WiFi, satellite, or othercellular or wireless technology. In other embodiments, the communicationinterface 250 may communicate through a wired connection and may be incommunication with one or more networks, such as Ethernet, token ring,USB, FireWire 1394, fiber optic, etc. In some embodiments, electronicdevice 200 comprises a single communication interface 250. In otherembodiments, electronic device 200 comprises two, three, four, or morecommunication interfaces. Thus, in some embodiments, electronic device200 can communicate with one or more components and/or devices throughone or more communication interfaces. In other embodiments, anelectronic device 200 may not comprise a communication interface 250.

In FIG. 2, the sensor 270 is in communication with the processor 210 andprovides sensor information to the processor 210. For example, sensor270 may provide one or more interactions to the processor 210. Thesensor 270 may provide an input signal indicating one or moreinteractions. As another example, sensor 270 can provide informationcorresponding to one or more interactions with electronic device 200 toprocessor 210. In some embodiments, the information the sensor 270provides to processor 210 corresponds to an interaction with the entireelectronic device 200, such as a user shaking the electronic device 200.In other embodiments, the information sensor 270 provides to processor210 corresponds to an interaction with a part of the electronic device200, such as a touch-sensitive display 230 or another suitable inputdevice.

The embodiment shown in FIG. 2 depicts a single sensor 270. In someembodiments, multiple sensors can be used. Additionally, a sensor may behoused in the same component as the other components of the electronicdevice 200 or in a separate component. For example, in some embodiments,the processor 210, memory 220, and sensor 270 are all comprised in anelectronic device 200, such as a portable music player, a portabletelephone, and/or a wearable device. In some embodiments, a sensor isplaced in component separate from another component that houses thememory and/or processor. For instance, a wearable sensor may be incommunication with the processor and memory or an electronic device viaa wired or wireless connection.

Sensor 270 may comprise any number and/or type of sensing components.For example, sensor 270 can comprise an accelerometer and/or gyroscope.A non-limiting list of examples of sensors and interactions is providedbelow:

TABLE 1 Exemplary Sensors and Conditions Sensor Interaction SensedAccelerometer Force in one, two, or three directions Altimeter AltitudeThermometer Ambient temperature; user body temperature Heart ratemonitor Heart rate of device user Skin resistance Skin resistance ofdevice user monitor Oxygen sensor Oxygen use of device user Audiosensor/ Ambient audio and/or audio generated microphone by device userPhotosensor Ambient light IR/Photosensor User eye movement, position,body temperature Hygrometer Relative humidity Speedometer VelocityPedometer/odometer Distance traveled Chronometer Time of day, dateWeight Mass or quantity of matter

Illustrative System

FIG. 3 illustrates a system diagram depicting illustrative computingdevices for visual processing of spectrograms to generate haptic effectsin an illustrative computing environment according to an embodiment. Thesystem 300 shown in FIG. 3 includes three electronic devices, 320-340,and a web server 350. Each of the electronic devices, 320-340, and theweb server 350 are connected to a network 310. In this embodiment, eachof the electronic devices, 320-340, is in communication with the webserver 350 through the network 310. Thus, each of the electronicdevices, 320-340, can send requests to the web server 350 and receiveresponses from the web server 350 through the network 310.

In an embodiment, the network 310 shown in FIG. 3 facilitatescommunications between the electronic devices, 320-340, and the webserver 350. The network 310 may be any suitable number or type ofnetworks or links, including, but not limited to, a dial-in network, alocal area network (LAN), wide area network (WAN), public switchedtelephone network (PSTN), a cellular network, a WiFi network, theInternet, an intranet or any combination of hard-wired and/or wirelesscommunication links. In one embodiment, the network 310 is a singlenetwork. In other embodiments, the network 310 may comprise two or morenetworks. For example, the electronic devices 320-340 may be connectedto a first network and the web server 350 may be connected to a secondnetwork and the first and the second network may be connected by a thirdnetwork. Numerous other network configurations would be obvious to aperson of ordinary skill in the art.

An electronic device may be capable of communicating with a network,such as network 310, and capable of sending and receiving information toand from another device, such as web server 350. For example, in FIG. 3,one electronic device 320 is a tablet computer. The tablet computer 320includes a touch-sensitive display and is able to communicate with thenetwork 310 by using a wireless communication interface card. Anotherdevice that may be an electronic device 330 shown in FIG. 3 is a desktopcomputer. The desktop computer 330 is in communication with a displayand is able to connect to the network 330 through a wired networkconnection. The desktop computer 330 may be in communication with anynumber of input devices such as a keyboard or a mouse. In FIG. 3, amobile phone is an electronic device 340. The mobile phone 340 may beable to communicate with the network 310 over a wireless communicationsmeans using Bluetooth, CDMA, TDMA, FDMA, GSM, WiFi, or other cellular orwireless technology.

A device receiving a request from another device may be any devicecapable of communicating with a network, such as network 310, andcapable of sending and receiving information to and from another device.For example, in the embodiment shown in FIG. 3, the web server 350 mayreceive a request from another device (e.g., one or more of electronicdevices 320-340) and may be in communication with network 310. Areceiving device may be in communication with one or more additionaldevices, such as additional servers. For example, web server 350 in FIG.3 may be in communication with another server. In an embodiment, a webserver may communicate with one or more additional devices to process arequest received from an electronic device. For example, web server 350in FIG. 3 may be in communication with a plurality of additionalservers, at least one of which may be used to process at least a portionof a request from any of the electronic devices 320-340. In oneembodiment, web server 350 may be part of or in communication with acontent distribution network (CDN).

One or more devices may be in communication with a data store. In FIG.3, web server 350 is in communication with data store 360. In someembodiments, data store 360 is operable to receive instructions from webserver 350 and/or other devices in communication with data store 360 andobtain, update, or otherwise process data in response to receiving theinstructions. In one embodiment, an electronic device, such as tabletcomputer 320, comprises and/or is in communication with a data store. Adata store, such as data store 360, may contain electronic content, suchas an eBook or magazine, data items, user accounts, metadata,information associated with predefined haptic effects, informationassociated with predefined events, associations between predefinedhaptic effects and predefined events, information associated withspectrogram characteristics, associations between spectrogramcharacteristics and predefined haptic effects, user interactions, userhistory, information regarding occurrences of events, default parametersfor one or more haptic effects, haptic profiles for one or moreoperating environments, one or more tactile models, minimum and/ormaximum parameters for a haptic effect, information regarding generatedpredefined haptic effects, interactions, parameters, parameteradjustments, correlations between interactions and parameteradjustments, correlations between parameter adjustments and profilesand/or operating modes, correlations between tactile models andinteractions, correlations between tactile models and haptic effects,correlations between tactile models and parameters, correlations betweenprofiles and/or operating modes and interactions, other informationusable to modify parameters of a haptic effect, information usable todetermine an interaction, other information, or a combination thereof.

Data store 360 shown in FIG. 3 can receive requests from web server 350and send responses to web server 350. For example, web server 350 mayreceive a request from tablet computer 320 for a predefined hapticeffect and a default intensity parameter. In response to receiving therequest from the tablet computer 320, web server 350 may query datastore 360 for the predefined haptic effect and the default intensityparameter for the predefined haptic effect. In response to receiving therequest from the web server 350, data store 360 may send the web server350 the predefined haptic effect and the default intensity parameter.The web server 350, can send the predefined haptic effect and thedefault intensity parameter to the tablet computer 320. The tabletcomputer 320 may modify the default intensity parameter for thepredefined haptic effect based at least in part on one or moreinteractions. For example, if one or more interactions indicate that agreater or otherwise more intense haptic effect should be output, thenthe tablet computer 320 may increase the intensity parameter above thedefault intensity parameter. Similarly, if one or more interactionsindicate that a lesser or otherwise less intense haptic effect should begenerated, then the table computer 320 may decrease the intensityparameter below the default intensity parameter. Numerous otherembodiments are disclosed herein and variations are within the scope ofthis disclosure.

Illustrative Method of Visual Processing of Spectrograms to GenerateHaptic Effects

FIG. 4 illustrates a flow chart directed to a method 400 of visualprocessing of spectrograms to generate haptic effects in accordance withan embodiment. The method 400 shown in FIG. 4 will be described withrespect to electronic device 200 shown in FIG. 2. In some embodiments,method 400 can be performed by one or more of the devices shown insystem 300 in FIG. 3. The method 400 shown in FIG. 4 will also bedescribed with respect to FIG. 5 which illustrates aspects of visualprocessing of spectrograms to generate haptic effects in accordance withan embodiment and with respect to FIG. 6 which illustrates aspects ofvisual processing of spectrograms to generate haptic effects inaccordance with an embodiment.

The method 400 begins in block 410 when a signal is received. Forexample, referring to FIG. 2, an audio signal may be received frommemory 220, another device in communication with processor 210 throughcommunication interface 250, and/or via sensor 270. As another example,referring to FIG. 3, in some embodiments tablet computer 320 receivesone or more signals, such as an audio signal, from web server 350 and/ordata store 360 through network 310.

One or more signals may be received from any number of components. Inone embodiment, a file, such as an audio or audio/video file, stored inmemory 220 of electronic device 200 is received by processor 210. Insome embodiments, a signal is received by processor 210 from anotherdevice through communication interface 250. For example, processor 210may receive an audio file and/or an audio/video file from a web serverthrough communication interface 250. As another example, processor 210can receive a continuous audio stream, such as streaming radio, from aserver through communication interface 250. In some embodiments,processor 210 receives a signal from sensor 270. For example, processor210 may receive an audio signal from a microphone connected with theelectronic device 200. As another example, a microphone may be disposedwithin housing 205 of the electronic device and the processor 210receives audio signals from the microphone. In other embodiments, one ormore sensor signals are disposed entirely or at least partially within ahousing of an electronic device. In one embodiment, processor 210receives audio information from a sensor of another device throughcommunication interface 250. For example, another electronic device mayhave a microphone and processor 210 receives audio information from themicrophone through communication interface 250 connected with a network,such as network 310 shown in FIG. 3. The one or more received signalscan be any number of signals including, but not limited to, an audiosignal, a video signal, an acceleration signal, a velocity signal, atemperature signal, another suitable signal, or a combination thereof.In various embodiments, one or more received signals can be any numberof signals for which one or more spectrograms can be generated. Numerousother embodiments are disclosed herein and variations are within thescope of this disclosure.

Referring back to method 400, once one or more signals are received 410,then method 400 proceeds to block 420. In block 420, one or morespectrograms are generated. For example, referring to FIG. 2, one ormore spectrograms may be generated by processor 210. As another example,referring to FIG. 3, one or more spectrograms may be generated bydesktop computer 330 and sent to server 350 through network 310 to beanalyzed. In one embodiment, tablet computer 320 receives an audio filefrom server 350 and generates one or more spectrograms to be analyzed bythe tablet computer 320.

One or more spectrograms can be generated in any number of ways. In oneembodiment, a Fourier transform of a time window of a time domain of asignal may be calculated and used to generate one or more spectrograms.Such an embodiment can product a three-dimensional matrix with onedimension representing time sections, another dimension representingfrequencies, and a third dimension representing Fourier transformvalues. These values may be used to generate a spectrogram, such as atwo-dimensional image with the horizontal axis representing time, thevertical axis representing frequency, and the color at a particularfrequency and time representing a Fourier transform value. For example,the top plot in FIG. 5 shows a two-dimensional representation of aspectrogram of a video clip corresponding to the time domainrepresentation of the audio signal shown in the bottom plot in FIG. 5.As shown in FIG. 5, the color intensity of the top plot represents theFFT magnitude for each time window. Furthermore, as shown in FIG. 5, thetop plot is a two-dimensional image and can be processed to determineone or more haptic effects as discussed herein.

One or more spectrograms can be generated and/or a generated spectrogramcan be divided into multiple spectrograms in any number of ways. In oneembodiment, a spectrogram is generated based at least in part on a timewindow of one or more received signals. For example, a time window of atime domain of a signal can be calculated and used to generate aspectrogram. In some embodiments, the time window can be any suitabletime window such as 1 millisecond, 10 milliseconds, 25 milliseconds, 50milliseconds, 75 milliseconds, 100 milliseconds, or any other suitabletime window. In some embodiments, a spectrogram is generated by at leastcalculating a Fourier transform for at least one segment of a signal,such as an audio signal and/or another suitable signal. One or moresegments can correspond to values of the signal, such as the audiosignal, in a respective time window. In some embodiments, one or moresegments of the signal used to generate a spectrogram at least partiallyoverlaps. In other embodiments, one or more segments of the signal usedto generate a spectrogram does not overlap with another segment of thesignal. Numerous other embodiments are disclosed herein and variationsare within the scope of this disclosure.

In some embodiments, a spectrogram is divided into multiple spectrogramsbased at least in part on frequency and/or time. In other embodimentsseparate spectrograms may be generated based at least in part onfrequency, time, and/or time windows. For example, FIG. 6 illustratesaspects of visual processing of spectrograms to generate haptic effectsin accordance with an embodiment. As shown in FIG. 6, the top image is aspectrogram of an audio signal displaying frequencies between 0 and 100Hz, the next image is a spectrogram of the same audio signal butcorresponding to frequencies between 100 and 1000 Hz, the third image isa spectrogram of the same audio signal but corresponding to frequenciesbetween 1000 and 5000 Hz, and the fourth image is a spectrogram of thesame audio signal but corresponding to frequencies between 500 and theNyquist frequency. Spectrograms can be generated for any number offrequencies or a previously-generated spectrogram can be divided intomultiple spectrograms for any number of frequencies.

In some embodiments, generating the spectrogram comprises generating oneor more two-dimensional images. A generated two-dimensional image maycorrespond to a first frequency range associated with the signal. Insome embodiments, the two-dimensional image is segmented into aplurality of two-dimensional images. One or more of the images in theplurality of two-dimensional images may comprise a respective frequencyrange. The respective frequency range may be a frequency range withinthe frequency range associated with the spectrogram. Each image in theplurality of two-dimensional images may correspond with a differentfrequency range than other image in the plurality of two-dimensionalimages. In some embodiments, the frequency range of a particular imagein the plurality of two-dimensional images partially overlaps with thefrequency range of another image in the plurality of two-dimensionalimage. In some embodiments, generating the spectrogram comprisesgenerating a plurality of spectrograms. One or more of the spectrogramsmay correspond with a frequency range associated with a received signal.In other embodiments, the spectrograms correspond with distinctfrequency ranges associated with the signal. Numerous other embodimentsare disclosed herein and variations are within the scope of thisdisclosure.

Referring back to method 400, once one or more spectrograms have beengenerated 420, then method 400 proceeds to block 430. In block 430, oneor more haptic effects are determined. For example, referring to FIG. 2,one or more haptic effects may be determined by processor 210 based atleast in part on one or more of the generated spectrograms. As anotherexample, referring to FIG. 3, one or more haptic effects may bedetermined by desktop computer 330 based at least in part on a databaseof characteristics that correspond with one or more haptic effects. Thedatabase of characteristics may be stored in the desktop computer 330and/or in one or more databases, such as database 360, in communicationwith server 350 and in communication with the desktop computer 330through network 310.

One or more haptic effects can be determined in any number of ways. Forexample, in one embodiment, a haptic effect is determined based at leastin part on one or more spectrograms. For instance, intensity variationsover time and/or frequency can be analyzed to determine one or morehaptic effects. As another example, image processing techniques can beapplied to one or more spectrograms to determine one of more hapticeffects. Such processing techniques include, but are not limited to,edge detection, thresholding, clustering, and/or color histogramestimation. In one embodiment, an edge detection technique, such asgradient estimation, is used to locate areas of transition betweencharacteristics in a spectrogram. In another embodiment, a thresholdingtechnique is used if a small number of gray or color levels is presentin a spectrogram. Thus, in some embodiments, a spectrogram can beconverted into a grayscale or a black and white image and a thresholdingtechnique can be applied to this image to determine one or more hapticeffects. In some embodiments, one or more spectrograms can bepartitioned into a given number of clusters that have contiguous pixelswith similar characteristics. The partitions may be analyzed todetermine one or more haptic effects. In other embodiments, hue,saturation, and lightness (HLS), hue, saturation, and value (HSV), hue,saturation, and intensity (HSI), and/or red, green, blue (RGB) valuesare analyzed in one or more spectrograms to identify regions withsimilar histograms. Numerous other embodiments are disclosed herein andvariations are within the scope of this disclosure.

In some embodiments, one or more haptic effects can be determined atleast in part by matching haptic effects with regions of one or morespectrograms. For example, if regions within one or more spectrogramsare spaced apart at regular intervals (e.g., in time), then a hapticeffect corresponding to a beat may be determined. Thus, in oneembodiment, beats in a received signal can be identified from one ormore spectrograms generated for the received signal and can be used tooutput haptic effects corresponding to the identified beats. As anotherexample, changes between one region and another region in one or morespectrograms can be used to determine a transition from onecharacteristic to another characteristic, such as a transition from onesound to another sound, in a received signal. In one embodiment, adetermination is made that a haptic effect should be output before adetermined transition from one characteristic to another characteristicin a received signal based at least in part on a transition between thecharacteristics in a spectrogram for the received signal. In anotherembodiment, a determination is made that a haptic effect should beoutput after a determined transition from one characteristic to anothercharacteristic in a received signal based at least in part on atransition between the characteristics in a spectrogram for the receivedsignal.

In some embodiments, determining a characteristic of the spectrogramcomprises applying at least one technique to the spectrogram to generatea plurality of regions. In some embodiments, a subset of the pluralityof regions is matched with at least one haptic effect. Determining ahaptic effect may comprise analyzing a subset of a plurality ofspectrograms for a characteristic corresponding to a predefined hapticeffect. If the characteristic is located within the subset of theplurality of spectrograms then the predefined haptic effect may beselected. Numerous other embodiments are disclosed herein and variationsare within the scope of this disclosure.

In some embodiments, one or more generated spectrograms for a signaltarget an entire frequency range or different frequency ranges. Forexample, the spectrogram shown in the top plot in FIG. 5 shows aspectrogram of an audio signal over all possible frequencies in thesignal (e.g., 0 Hz to the Nyquist frequency in this figure). In someembodiments, a spectrogram, such as the spectrogram shown in FIG. 5,provides information that can be used to determine one or more hapticeffects. In other embodiments, more detailed information in two or morefrequency ranges are used to determine one or more haptic effects

For example, as shown in FIG. 6, the top image is a spectrogram of anaudio signal displaying frequencies between 0 and 100 Hz, the next imageis a spectrogram of the same audio signal but corresponding tofrequencies between 100 and 1000 Hz, the third image is a spectrogram ofthe same audio signal but corresponding to frequencies between 1000 and5000 Hz, and the fourth image is a spectrogram of the same audio signalbut corresponding to frequencies between 500 and the Nyquist frequency.As can be seen in FIG. 6, spectrograms based at least in part ondifferent frequency ranges provide different information that can beused to determine one or more haptic effects. For example, at varioustimes, each frequency band shown in FIG. 6 contains different intensityinformation as demonstrated by the four circles, each covering the sameperiod of time but at different frequency bands. In some embodiments,one or more of these frequency bands (e.g., frequency ranges) areanalyzed for one or more characteristics. One or more haptic effectscorresponding to one or more characteristics may be determined byanalyzing any or all of the frequency ranges of one or morespectrograms.

One or more frequency ranges of one or more spectrograms may be analyzedfor characteristic(s) that correspond with one or more predeterminedhaptic effects. For example, a haptic effect may be selected if aparticular characteristic is located in a particular frequency range ofone or more spectrograms. Thus, in some embodiments, a plurality offrequency ranges of one or more spectrograms for a received signal areanalyzed. In one embodiment, an electronic device 200 that receives asignal generates one or more spectrograms for one or more frequencyranges and analyzes at least some of the spectrograms forcharacteristics corresponding to haptic effects. For example, electronicdevice 200 may have a library of predefined haptic effects thatcorrespond with characteristics. In this embodiment, one or moregenerated spectrograms can be analyzed to determine if thecharacteristic is present in the spectrograms. In other embodiments,electronic device 200 may comprise or be in communication with adatabase of predefined haptic effects and/or a database of predefinedcharacteristics and/or associations between predefined haptic effectsand predefined characteristics. Numerous other embodiments are disclosedherein and variations are within the scope of this disclosure.

In response to determining that a spectrogram has a characteristic, oneor more haptic effects corresponding to the characteristic may bedetermined based at least in part on the library of predefined hapticeffects and/or correlations between predefined haptic effects andparticular characteristics. In some embodiments, the timing for one ormore determined haptic effects is based at least in part on a time whenthe characteristic is found in one or more spectrograms. For example, adetermination may be made that a haptic effect corresponding to acharacteristic should be output before the characteristic occurs (e.g.,5 ms before the characteristic, 50 ms before the characteristic, or anyother suitable time before the characteristic occurs). As anotherexample, a determination can be made that a haptic effect correspondingto a characteristic should be output when the characteristic occurs. Inother embodiments, a determination is made that a haptic effectcorresponding to a characteristic should be output before, when, during,or after a characteristic occurs. In some embodiments, a determinationis made that a haptic effect should be output during a transition fromone characteristic to another characteristic. Numerous other embodimentsare disclosed herein and variations are within the scope of thisdisclosure.

One or more of the frequency ranges may be distinct from other analyzedfrequency ranges and can be analyzed to determined one or morecharacteristics and/or one or more haptic effects. For example,referring again to FIG. 6, in one embodiment each of the four frequencyranges are independently analyzed for one or more characteristics and/orone or more haptic effects. Thus, in FIG. 6, a spectrogram of afrequency band of 100 to 1000 Hz is analyzed separately from aspectrogram of a frequency band of 1000 to 5000 Hz. In some embodiments,once two or more of the frequency bands have been independently analyzedto determine haptic effects, a single haptic track is determined basedat least in part on the determined haptic effects for the independentlyanalyzed frequency bands.

In other embodiments, two or more haptic tracks are determined based atleast in part on the independently analyzed frequency bands. Forexample, referring back to FIG. 6, a first haptic track may bedetermined based at least in part on a spectrogram of a frequency bandof 100 to 1000 Hz and a second haptic track may be determined based atleast in part on a spectrogram of a frequency band of 1000 to 5000 Hz.In other embodiments, some haptic effect tracks based on two or morefrequency bands are combined into a single haptic effect track and otherhaptic effect tracks based on another frequency band is maintained as aseparate haptic effect track. Thus, one or more haptic effect tracks canbe generated based at least in part on one or more generatedspectrograms. In some embodiments two or more spectrograms are analyzedindependently and then combined into a haptic effect track. In thisembodiment, additional haptic effects can be added to and/or subtractfrom a combined haptic effect track based at least in part on hapticeffects determined for each spectrogram. For example, if bothspectrograms indicate that a particular haptic effect should be outputbut the timing for the haptic effect does not match exactly, adetermination may be made to output the haptic effect only once at oneof the two times, an average of the two times, etc. As another example,if both spectrograms indicate that haptic effects should be output atthe same or a similar time, then a single haptic effect may bedetermined for the combined track that has a larger magnitude thaneither of the individually determined haptic effects. Numerous otherembodiments are disclosed herein and variations are within the scope ofthis disclosure.

In one embodiment, a user is able to select one or more haptic effecttracks. For example, as discussed above, two or more haptic effecttracks can be determined based at least in part on one or more frequencybands of one or more spectrograms for a received signal. In oneembodiment, two or more of the determined haptic effect tracks arepresented to a user, such as in a user interface, and the user canselect which haptic output track or tracks to use. Thus, in oneembodiment, a first haptic effect track corresponds with haptic effectsdetermined in a first frequency range and a second haptic effect trackcorresponds with haptic effects determined in a second frequency range.In other embodiments, two, three, four, five, or more haptic effecttracks are presented to a user based at least in part on one or morefrequency ranges. In some embodiments, a user can choose to combine twoor more of the haptic tracks to create a single haptic output track.Numerous other embodiments are disclosed herein and variations arewithin the scope of this disclosure.

Referring back to method 400, once one or more haptic output signalshave been determined 430, the method 400 proceeds to block 440. In block440, one or more haptic output signals are generated. For example,referring to FIG. 2, electronic device 200 may generate one or morehaptic output signals. As another example, referring to FIG. 3, tabletcomputer 320 may generate one or more haptic output signals. In oneembodiment, tablet computer 320 may send one or more generated hapticoutput signals to another device, such desktop computer 330 shown inFIG. 3. In an embodiment, one or more of the haptic output signals isbased at least in part on a modified haptic effect. For example, ahaptic output signal may be configured to cause one or more hapticoutput devices to output a modified haptic effect. Thus, for example, ifan intensity parameter corresponding to a determined haptic effect hasbeen modified, then the haptic output signal may be configured to causea haptic output device to output a haptic effect that has an intensitycorresponding to the modified intensity parameter. Numerous otherembodiments are disclosed herein and variations are within the scope ofthis disclosure.

In some embodiments, the processor 210 generates a single signal when anevent occurs. For example, in one embodiment, the processor 210generates a signal configured to cause a haptic output device, such ashaptic output device 240 or haptic output device 260, to output a hapticeffect. The haptic effect may indicate that an object is currentlydisplayed on the display 230, that an object is about to be displayed onthe display 230, that an object is approaching, that an event hasoccurred, that an event is about to occur, or a combination thereof.

In other embodiments, the processor 210 generates two, three, or moresignals. For example, in one embodiment, the processor 210 generates afirst signal configured to cause a first haptic effect and a secondsignal configured to cause a second haptic effect. In some embodiments,the processor 210 generates a different signal for each event thatoccurs. In various embodiments, the processor 210 generates one or moresignals configured to cause the touch-sensitive display 230, thecommunication interface 250, the haptic output device 240, the hapticoutput device 260, the sensor 270, other components of the device 200,other components of devices in communication with the device 200, or acombination thereof to output one or more of the generated signals, suchas a video signal, audio signal, haptic output signal, and/or acommunication signal. For example, in one embodiment, the processor 210generates a signal when the event occurs where the signal is configuredto cause a haptic output device in another device to cause a hapticeffect. In one embodiment, the processor 210 sends the signal to theother device through the communication interface 250.

In one embodiment, a generated signal includes a command for a device orcomponent to perform a specified function, such as to output a hapticeffect or transmit a message to a remote device. In another embodiment,a generated signal includes parameters which are used by a device orcomponent receiving the command to determine a response or some aspectof a response. Parameters may include various data related to, forexample, magnitudes, frequencies, durations, or other parameters that ahaptic output device can use to determine a haptic effect, output ahaptic effect, or both. For example, in one embodiment, the processor210 generates a signal configured to cause haptic output device 240 tooutput a haptic effect. In such an embodiment, the signal may include apressure parameter that the haptic output device 240 uses to determinethe intensity of the haptic effect to output. For example, according toone embodiment, the larger the pressure parameter the haptic outputdevice 240 receives, the more intense the haptic effect that is output.

In one embodiment, an intensity parameter is used by a haptic outputdevice to determine the intensity of a haptic effect. In thisembodiment, the greater the intensity parameter, the more intense thehaptic effect that is output. In one embodiment, the intensity parameteris based at least in part on sensor information, such as speed,direction, etc., of a remotely controllable device when an event occurs.Thus, according to one embodiment, a larger intensity parameter is sentto a haptic output device when an event occurs while the remotelycontrollable device is travelling at a faster speed than when an eventoccurs while the remotely controllable device is travelling at a slowerspeed. A signal may include data that is configured to be processed by ahaptic output device, display, communication interface, sensor, or othercomponents of a device or in communication with a device in order todetermine an aspect of a particular response.

Referring back to method 400, once one or more haptic output signalshave been generated 440, the method 400 proceeds to block 450. In block450, one or more generated haptic output signals are output to one ormore haptic output devices. For example, referring to FIG. 2, one ormore generated haptic output signals may be output to haptic outputdevice 240 and/or haptic output device 260. As another example,referring to FIG. 3, one or more haptic output signals generated bydesktop computer 330 may be output to one or more haptic output devicesin tablet computer 320 through network 310. In one embodiment, agenerated haptic output signal is sent to one haptic output device. Inother embodiments, a generated haptic output signal is sent to two,three, four, or more haptic output devices. In some embodiments, two,three, four, or more generated haptic output signals are sent to ahaptic output device. In other embodiments, two, three, four, or moregenerated haptic output signals are set to two, three, four, or morehaptic output devices. Numerous other embodiments are disclosed hereinand variations are within the scope of this disclosure.

In various embodiments, the processor 210 may output one or moregenerated signals to any number of devices. For example, the processor210 may output one signal to the communication interface 250. In oneembodiment, the processor 210 may output one generated signal to thetouch-sensitive display 230, another generated signal to thecommunication interface 250, and another generated signal to the hapticoutput device 260. In other embodiments, the processor 210 may output asingle generated signal to multiple components or devices. For example,in one embodiment, the processor 210 outputs one generated signal toboth haptic output device 240 and haptic output device 260. In anotherembodiment, the processor 210 outputs one generated signal to hapticoutput device 240, haptic output device 260, and communication interface250. In still another embodiment, the processor 210 outputs onegenerated signal to both haptic output device 240 and haptic outputdevice 260 and outputs a second generated signal to the touch-sensitivedisplay 230.

As discussed above, the processor 210 may output one or more signals tothe communication interface 250. For example, the processor 210 mayoutput a signal to the communication interface 250 instructing thecommunication interface 250 to send data to another component or devicein communication with the device 200. In such an embodiment, thecommunication interface 250 may send data to the other device and theother device may perform a function such as updating a displayassociated with the other device or the other device may output a hapticeffect. Thus, in some embodiments, a second device may output a hapticeffect based at least in part upon an interaction with a first device incommunication with the second device. In other embodiments, a seconddevice may perform any number of functions such as, for example,updating a display associated with the second device or outputting asound to a sensor associated with the second device based at least inpart on an interaction with a first remote control 200.

In various embodiments, after the processor 210 outputs a signal to acomponent, the component may send the processor 210 a confirmationindicating that the component received the signal. For example, in oneembodiment, haptic output device 260 may receive a command from theprocessor 210 to output a haptic effect. Once haptic output device 260receives the command, the haptic output device 260 may send aconfirmation response to the processor 210 that the command was receivedby the haptic output device 260. In another embodiment, the processor210 may receive completion data indicating that a component not onlyreceived an instruction but that the component has performed a response.For example, in one embodiment, haptic output device 240 may receivevarious parameters from the processor 210. Based on these parametershaptic output device 240 may output a haptic effect and send theprocessor 210 completion data indicating that haptic output device 240received the parameters and outputted a haptic effect. Numerous otherembodiments are disclosed herein and variations are within the scope ofthis disclosure.

General

While the methods and systems herein are described in terms of softwareexecuting on various machines, the methods and systems may also beimplemented as specifically-configured hardware, such asfield-programmable gate array (FPGA) specifically configured to executethe various methods. For example, embodiments can be implemented indigital electronic circuitry, or in computer hardware, firmware,software, or in a combination thereof. In one embodiment, a device maycomprise a processor or processors. The processor comprises acomputer-readable medium, such as a random access memory (RAM) coupledto the processor. The processor executes computer-executable programinstructions stored in memory, such as executing one or more computerprograms for editing an image. Such processors may comprise amicroprocessor, a digital signal processor (DSP), anapplication-specific integrated circuit (ASIC), field programmable gatearrays (FPGAs), and state machines. Such processors may further compriseprogrammable electronic devices such as PLCs, programmable interruptcontrollers (PICs), programmable logic devices (PLDs), programmableread-only memories (PROMs), electronically programmable read-onlymemories (EPROMs or EEPROMs), or other similar devices.

Such processors may comprise, or may be in communication with, media,for example computer-readable media, that may store instructions that,when executed by the processor, can cause the processor to perform thesteps described herein as carried out, or assisted, by a processor.Embodiments of computer-readable media may comprise, but are not limitedto, an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The foregoing description of some embodiments of the invention has beenpresented only for the purpose of illustration and description and isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Numerous modifications and adaptations thereof will beapparent to those skilled in the art without departing from the spiritand scope of the invention.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, operation, or other characteristicdescribed in connection with the embodiment may be included in at leastone implementation of the invention. The invention is not restricted tothe particular embodiments described as such. The appearance of thephrase “in one embodiment” or “in an embodiment” in various places inthe specification does not necessarily refer to the same embodiment. Anyparticular feature, structure, operation, or other characteristicdescribed in this specification in relation to “one embodiment” may becombined with other features, structures, operations, or othercharacteristics described in respect of any other embodiment.

That which is claimed is:
 1. A method comprising: receiving a signal;generating a spectrogram based at least in part on the signal;determining a haptic effect based at least in part on the spectrogram;generating a haptic output signal based at least in part on the hapticeffect, the haptic output signal configured to cause a haptic outputdevice to output the haptic effect; and outputting the haptic outputsignal.
 2. The method of claim 1, wherein the signal comprises an audiosignal.
 3. The method of claim 2, wherein the signal further comprises avideo signal.
 4. The method of claim 1, wherein the signal comprises atleast one of an acceleration signal, a velocity signal, or a temperaturesignal.
 5. The method of claim 2, wherein generating the spectrogramcomprises: calculating a Fourier transform for at least one segment ofthe audio signal, each segment corresponding to values of the audiosignal in a respective time window.
 6. The method of claim 5, whereinthe at least one segment comprises a first segment and a second segment,the second segment at least partially overlapping with the firstsegment.
 7. The method of claim 5, wherein the at least one segmentcomprises a first segment and a second segment, the second segment notoverlapping with the first segment.
 8. The method of claim 5, whereinthe at least one segment comprises a first segment and the respectivetime window of the first segment is less than 75 milliseconds.
 9. Themethod of claim 1, wherein generating the spectrogram comprises:generating a two-dimensional image corresponding to a first frequencyrange associated with the signal.
 10. The method of claim 9, whereingenerating the spectrogram further comprises: segmenting thetwo-dimensional image into a plurality of two-dimensional images, eachimage in the plurality of two-dimensional images comprising a respectivefrequency range within the first frequency range.
 11. The method ofclaim 10, wherein a first respective frequency range of a firsttwo-dimensional image in the plurality of two-dimensional images isdifferent than the respective frequency ranges of other two-dimensionalimages in the plurality of two-dimensional images.
 12. The method ofclaim 11, wherein the first respective frequency partially overlaps witha second respective frequency range of a second two-dimensional image inthe plurality of two-dimensional images.
 13. The method of claim 1,wherein generating the spectrogram comprises: generating a plurality ofspectrograms, each spectrogram corresponding with a respective distinctfrequency range associated with the signal.
 14. The method of claim 13,wherein determining the haptic effect comprises: analyzing a subset ofthe plurality of spectrograms for a characteristic corresponding to apredefined haptic effect; and in response to locating the characteristiccorresponding to the predefined haptic effect, selecting the predefinedhaptic effect.
 15. The method of claim 1, wherein determining the hapticeffect comprises: for each of a plurality of distinct frequency ranges:analyzing the spectrogram over that distinct frequency range for acharacteristic; determining a haptic effect corresponding to thecharacteristic; and in response to locating the characteristic in thespectrogram, selecting the haptic effect.
 16. The method of claim 1,wherein determining the haptic effect comprises: determining acharacteristic of the spectrogram; and determining a haptic effectcorresponding to the characteristic.
 17. The method of claim 16, whereindetermining the characteristic of the spectrogram comprises: for each ofa plurality of predefined characteristics, each predefinedcharacteristic corresponding with at least one predefined haptic effect:determining whether the spectrogram comprises that predefinedcharacteristic; and in response to determining that the spectrogramcomprises that predefined characteristic, selecting the at least onepredefined haptic effect corresponding with that predefinedcharacteristic as the determined haptic effect.
 18. The method of claim16, wherein the spectrogram comprises a time dimension, and whereindetermining the characteristic of the spectrogram comprises determininga pattern within the time dimension of the spectrogram.
 19. The methodof claim 15, wherein the spectrogram comprises a frequency dimension,and wherein determining the characteristic of the spectrogram comprisesdetermining a pattern within the frequency dimension of the spectrogram.20. The method of claim 16, wherein determining the characteristic ofthe spectrogram comprises: determining a transition between a firstcharacteristic and a second characteristic in the spectrogram.
 21. Themethod of claim 16, wherein determining the characteristic of thespectrogram comprises: applying at least one technique to thespectrogram to generate a plurality of regions, the at least onetechnique comprising an edge detection, a threshold detection, aclustering detection, or a color histogram cluster detection; andmatching a subset of the plurality of regions with at least one hapticeffect.
 22. A computer-readable medium comprising program code for:receiving a signal; generating a spectrogram based at least in part onthe signal; determining a haptic effect based at least in part on thespectrogram; generating a haptic output signal based at least in part onthe haptic effect, the haptic output signal configured to cause a hapticoutput device to output the haptic effect; and outputting the hapticoutput signal.
 23. A system, comprising: an input device; a hapticoutput device; and a processor in communication with the input deviceand the haptic output device, the processor configured for: receiving asignal from the input device; generating a spectrogram based at least inpart on the signal; determining a haptic effect based at least in parton the spectrogram; generating a haptic output signal based at least inpart on the haptic effect, the haptic output signal configured to causethe haptic output device to output the haptic effect; and outputting thehaptic output signal to the haptic output device.